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ABSTRACT 


This  paper  presents  the  development  of  scoring  functions  for  use  in  conjunction 
with  standard  multiple-choice  items.  In  addition  to  the  usual  indication  of  the  correct 
alternative,  the  method  requires  that  the  examinee  indicate  his  personal  probability  of  the 
correctness  of  his  response.  Both  linear  and  quadratic  polynomial  scoring  functions  are 
examined  for  suitability.  Unique  quadratic  scoring  functions  are  found  such  that  a  score 
of  zero  is  assigned  when  complete  uncertainty  is  indicated.  Furthermore,  the  examinee 
can  expect  to  do  best  if  he  reports  his  personal  probability  accurately.  A  table  of  simple 
integer  approximations  to  the  scoring  function  is  supplied. 


SUMMARY 


Boldt,  R.F.  A  simple  confidence  testing  format.  AFHRL-TR-71-31.  Lowry  AFB,  Colo.:  Technical  Training 

Division,  Air  Force  Human  Resources  Laboratory,  July  1971. 

Problem 

Dissatisfaction  in  some  quarters  with  the  multiple-choice  test  format  leads  to  a  search  for  some  other 
means  of  testing.  Confidence  testing  is  one  that  has  been  proposed;  however,  the  formats  used  have  seemed 
overly  demanding  on  examinees  or  on  scoring  facilities.  A  method  is  sought  which  is  a  compromise  of 
simplicity  and  practicality  for  the  examinee  and  the  scorer.  Also,  the  method  should  encourage  the 
examinee  to  reflect  accurately  his  personal  probability  of  correctness  of  his  response. 

Approach 

In  the  method  developed,  the  examinee  indicates  which  response  alternative  he  thinks  is  correct  and, 
on  a  scale  ranging  from  the  reciprocal  of  the  number  of  alternatives  to  one,  he  indicates  how  confident  he  is 
that  his  choice  is  correct.  Two  scoring  functions  are  chosen,  one  for  use  when  his  choice  is  correct,  the 
other  for  when  his  choice  is  incorrect.  Both  linear  and  quadratic  polynomial  functions  were  examined,  with 
the  constants  determined  by  the  enforcement  of  desirable  properties  of  the  functions.  These  properties 
were  that  the  expected  score  should  be  at  a  maximum  when  the  examinee’s  res;.  r  nse  is  equal  to  his  personal 
probability  of  being  correct  and  that  the  score  indicating  complete  uncertainty  should  equal  zero  whether 
the  right  or  wrong  response  is  made.  It  was  also  useful  to  choose  as  an  arbitrary  scot*  the  value  of  unity 
when  the  examinee  indicated  certainty  of  the  correctness  of  the  correct  response. 

Results 

It  was  found  that  the  linear  function  best  conforming  to  the  required  conditions  is  the  usual  formula 
score.  Therefore,  use  of  linear  scoring  functions  does  not  lead  to  improvement  of  the  current  system. 
However,  when  quadratic  scoring  functions  are  used,  there  are  unique  scoring  functions  for  correct  and 
incorrect  choices  which  satisfy  the  required  conditions.  The  coefficients  of  these  functions  depend  on  the 
number  of  alternatives,  and  a  table  supplying  values  of  the  scoring  functions  is  supplied  for  true-false 
items,  for  three-,  four-,  and  five-alternative  items,  and  for  free  response  items.  Further,  a  table  of  simpie 
approximations  to  these  values  of  the  scoring  functions  is  presented.  The  table  uses  simple  integer  values 
and  requires  only  that  the  examinee  make  approximate  indications  of  his  degree  of  certainty  of  the 
correctness  of  his  choice. 

Condusirms 

A  response  format  for  confidence  testing  is  presented,  and  scoring  functions  are  developed.  The 
scoring  required  would  not  be  usable  with  standard  scoring  equipment  except  when  supplemented  by 
addition  digital  processing.  However,  the  scoring  would  allow  the  examinee  to  indicate  uncertainty  when  he 
feels  it  and  encourage  him  to  give  an  accurate  expression  of  his  personal  probability  of  correctness  of  the 
alternative  chosen. 

This  summary  was  prepared  by  Wayne  S.  Sellman,  Technical  Training  Division.  Air  Force  Human 
Resources  Laboratory. 
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A  SIMPLE  CONFIDENCE  TESTING  FORMAT 


I.  INTRODUCTION 

Test  takers  and  test  developers  have  long  been 
aware  that  multiple-choice  item  format  has  certain 
presumed  deficiencies.  Among  these  is  the 
examinee’s  presumed  anxiety  generated  by  the 
need  to  indicate  either-or  conclusions  about  the 
correctness  of  the  item.  Further,  the  scorer  is  un¬ 
able  to  differentiate  between  answers  which  are  a 
product  of  knowledge  and  those  which  are  largely 
a  product  of  uncertainty.  While  it  is  true  that  the 
traditional  methods  work  and  have  not  as  yet  been 
improved  upon  in  a  way  that  demonstrably  up¬ 
grades  their  utility  to  the  score  user,  one  might 
still  be  willing  to  accept  some  additional 
complication  in  mass  processing  if  the  testing 
process  could  be  made  more  palatable  to  the 
examinee.  One  way  to  do  this  is  to  make  some 
provision  for  the  test  taker  to  communicate  the 
fact  that  he  is  uncertain  to  sor  e  extent  of  the 
correctness  of  the  response  that  he  is  making.  In 
this  way  and  with  a  reasonable  scoring  procedure, 
one  can  reassure  the  examinee  that  hesitant 
choices  among  responses  will  not  incur  large  score 
differences.  Thus,  the  intensity  of  the  conflict  en¬ 
countered  in  this  risky  decision  situation  should  be 
reduced,  and  the  testing  process  should  become 
rather  more  comfortable.  This,  at  least,  is  one  kind 
of  rationale  for  allowing  the  test  taker  to  com¬ 
municate  his  degree  of  uncertainty  about  his 
response. 

Various  ways  of  allowing  for  uncertainty  have 
been  made  ranging  from  the  garden  variety  scoring 
formula,  which  merely  eliminates  the  advantage  to 
a  guesser  if  a  rights-only  score  is  used,  to  the  more 
elaborate  subjective  probability  methods  intro¬ 
duced  by  de  Finetti  (1965),  who  requires  that  a 
scoring  method  oblige  the  examinee  “to  reveal  his 
true  beliefs,  because  any  falsification  will  turn  out 
to  be  to  his  disadvantage  ”  Stanley  (1968)  has 
described  a  variety  of  methods  allowing  for  un¬ 
certainty  including  those  where  the  main 
motivation  is  to  eliminate  advantages  due  to 
guessing.  These  methods  apparently  do  not  always 
yield  gains  in  reliability  and  do  nothing  for  the 
expression  of  degrees  of  confidence.  Confidence 
testing  has  been  discussed  by  Lord  and  Novick 
(1968),  and  studies  of  effects  on  reliability  nave 
been  summarized  by  Echtemacht  (1971).  How¬ 
ever,  a  method  suggested  by  Dressel  and  Schmid 
(19S3)  is  one  which  is  virtually  identical  with  that 
favored  in  this  paper.  A  forerunner  of  the  Dressel- 


Schmid  format  was  introduced  by  Hevner(1932), 
and  work  using  formats  highly  similar  to  that  of 
Dressel  and  Schmid  was  done  by  Soderquist 
(1936),  Wiley  and  Trimble  (1936),  Swineford 
(1938,  1941),  Gritten  and  Johnson  (1941),  and 
Frederiksen,  Jensen,  and  Beaton  (1968).  These 
studies  have  been  discussed  by  Stanley  (1968)  and 
Echternacht  (1971).  The  procedures  required  the 
examinee  to  mark  the  correct  alternative  and,  in 
addition,  to  assign  a  confidence  weight  (ranging 
from  one  to  four  in  the  case  of  Dressel  and 
Schmid)  in  accordance  with  his  degree  of  certainty 
as  to  the  correctness  of  the  choice.  To  anticipate 
later  development  in  this  paper,  it  may  be  noted 
that  in  a  sense  the  present  paper  presents  a  scoring 
rationale  and  weighting  scheme  for  the  Dressel- 
Schmid  confidence  format,  based  on  modern 
notions  of  subjective  probability.  It  should  be 
understood  that  the  author  is  not  endorsing  the 
uncritical  acceptance  of  confidence  testing 
practices.  Confidence  testing  has  its  probable 
drawbacks,  some  of  which  are  discussed  in  the  Iasi 
section  of  this  paper.  What  is  intended  is  that  the 
use  of  the  confidence  testing  be  made  easy,  while 
still  retaining  the  desirable  requirement  of  de 
Finetti  as  enunciated  in  the  discussion  to  follow. 

Shuford,  Albert,  and  Massengill  (1966)  have 
defined  a  “reproducing  scoring  system”  in  the 
spirit  of  de  Finetti  as  follows:  let  be  a 

function  of  the  vector,  R,  of  responses  to  a 
multiple-choice  item  when  alternative  h  is  the 
correct  one,  and  let  p.  be  the  test  taker’s  personnel 
probability  that  the  ith  response  is  the  correct  one. 
In  this  vein,  R  is  a  vector  with  non-negative 
elements  r.  which  sum  to  unity,  and  the  p.  are  also 
non-negative  and  sum  to  unity.  The  distinction  is 
made  that  R  is  the  vector  of  responses  actually 
made  which  may  or  may  not  correspond  to  the 
p’s  This  lack  of  correspondence  might  arise 
through  some  idiosyncratic  notions  about  test 
taking  strategies;  the  intent  of  the  scoring  system 
is  to  produce  a  situation  in  which  the  subject  can 
do  his  best  by  revealing  the  p's  as  accurately  as  he 
can.  This  is  to  be  done  by  taking  as  an  objective 
function  the  examinee’s  expected  score,  S.  with 
respect  to  his  own  personal  probability,  and  choos¬ 
ing  if  so  that  S  is  at  a  maximum  when  the  r’s  equal 
the  corresponding  p’s.  That  is,  choose  so  that 

S=Xph<fih(R) 

n 


l 


is  at  a  maximum  when  rh=Ph  for  all  admissible  sets 
of  p's,  and  subject  to  constraints  that  the  A  must 
be  non-negative  and  must  sum  to  unity.  Such  a 
scoring  system  is  called  by  Shuford,  Albert,  and 
Massengill  a  “reproducing  scoring  system”  because 
if  the  examinee  dees,  in  fact,  knowingly  behave  so 
as  to  maximize  S,  his  responses  will  reproduce  his 
subjective  probabilities. 

Note  that  the  functions  <fh  have  as  arguments 
the  elements  of  the  vector  R  and  hence  require  the 
recording  of  a  response  for  each  alternative.  Thus, 
the  task  of  the  examinee  is  to  choose  for  each  item 
a  vector,  R,  by  estimating  the  relative  strength  of 
his  subjective  attitudes  toward  the  alternatives  or 
according  to  some  personal  strategy.  This  task  may 
be  too  difficult  for  the  examinee,  and  may  be  care¬ 
lessly  done,  and  also  may  be  prohibitively 
expensive  to  score.  Hence,  the  simplicity  of  the 
Dressel-Schmid  format,  together  with  a  rationale 
using  subjective  prohamuty  notions  to  develop  the 
reproducing  property,  is  appealing.  Admittedly, 
the  Dressel-Schmid  format  will  not  be  fully 
reproducing  since  the  entire  vector  R  is  not 
developed-only  the  largest  element  in  R  is 
recorded;  therefore,  the  term  “quasi-reproducing” 
is  used  subsequently ,  referring  to  the  reproduction 
by  the  response  made  of  the  corresponding  under¬ 
lying  subjective  probability.  The  utility  of  the 
format  is  also  limited  in  that  its  adoption  over  the 
standard  formula  score  would  not  be  expected  to 
yield  more  than  minor  increases  in  reliability.  Its 
main  advantage  seems  to  the  author  to  be  its 
improved  credibility  and  the  attractiveness  of  the 
scoring  rationale;  i.e.,  the  situation  is  structured  so 
that  the  optimum  strategy  is  the  honest  expression 
of  the  answer  and  the  examinee's  confidence  in  its 
correctness.  It  is  felt  that  there  are  situations, 
particularly  those  in  which  test  anxiety  seems 
high,  where  these  advantages  may  be  compelling. 

The  present  paper  is  concerned  with  a  simplifi¬ 
cation  wherein  the  examinee  rates  his  response  to 
only  one  alternative,  the  alternative  rated 
indicating  his  choice  of  the  best  alternative  and  the 
rating  indicating  his  degree  of  confidence  in  that 
alternative  only.  The  response  will  be  scored  on 
whether  the  correct  alternative  was  marked  and 
how  confident  the  examinee  is  in  his  choice.  One 
would  like  a  scoring  scheme  in  which  wrong 
opinions  confidently  expressed  incur  large 
penalties,  frank  guesses  or  near  guesses  are  only 
mildly  punished  or  rewarded  if  at  all,  and  con¬ 
fidently  expressed  correct  opinions  ate  greatly 
rewarded.  Two  scoring  functions  will  be  used:  one 
if  the  correct  alternative  is  marked  and  another  if 


the  incorrect  alternative  is  marked.  Both  will  be 
monotonic  functions  of  the  level  of  confidence 
expressed,  and  it  will  turn  out  that  the  scoring 
function  for  the  correct  alternative  will  be 
monotonically  increasing  while  the  scoring  func¬ 
tion  for  an  incorrect  alternative  response  will  be 
monotonically  decreasing. 

If  the  level  of  confidence  recorded  is  x,  fix)  is 
the  scoring  function  if  the  correct  alternative  is 
marked,  g(x)  is  the  scoring  function  if  an  incorrect 
alternative  is  marked,  and  p  is  the  examinee’s 
subjective  probability  that  the  response  he  marked 
is,  in  fact,  correct;  then  the  objective  function 
becomes 

s=pyW+(i-p)*(*) 

and  one  wishes  to  choose  /  and  g  so  that  S  is  at  a 
maximum  if  x  equals  p  for  all  admissible  p. 
Various  constraints  can  be  imposed  on  the/ and g 
yielding  different  scoring  functions.  In  this  paper, 
linear  and  quadratic  functions  will  be  examined.  If 
more  requirements  seem  needed,  higher  order 
polynomials  could  be  adopted. 


11.  THE  LINEAR  CASE 

Assume 

/(■*)  =ax  +  b  and  g  (jr)  =  Ax  +  B. 

Then 

S  =  p  (ax  +  b)  +  (1  -  p)  {Ax  +  B). 

Since  in  this  case  S  is  linear  in  x,  it  follows  that  x 
should  take  on  an  extreme  value  because  the  func¬ 
tion  S  has  no  relative  minimum  in  the  interval  zero 
to  one.  Hence,  it  is  not  possible  to  get  a  quasi- 
reproducing  scoring  system  with  a  linear  scoring 
function  in  the  “pick  one”  format.  To  avoid 
forcing  the  examinee  to  express  certainty  when  he 
does  not  feel  so  certain,  set  both  a  and  A  equal  to 
zero.  Further,  set  S  equal  to  zero  when  p  is  one 
divided  by  the  number  of  alternatives  (the 
examinee  has  no  preferred  answer)  because  it 
seems  reasonable  to  have  an  expected  score  equal 
to  the  omit  score  when  uncertainty  prevails.  Omits 
will  be  given  a  zero,  so 

S  =  (6/ft)+(fc-l)B/*  =  0 
and 

f>=  — (fr  -  \)P 

where  k  is  the  numbei  of  alternatives.  If  b  is  taken 
as  positive,  the  examinee  will  respond  to  that  alter¬ 
native  for  which  his  subjective  probability  is  the 
highest  since  he  will  have  the  most  to  gain  (“good” 


scores  on  S  are  defined  as  being  in  the  positive 
direction).  Since  the  response  made  is  the  one  with 
the  highest  subjective  probability  and  since  the 
subjective  probabilities  must  add  up  to  one,  it 
follows  that  the  examinee  who  marks  the  answer 
with  a  confidence  of  1  /k  is  completely  uncertain. 
That  is,  if  the  highest  of  a  set  of  p’s  is  less  than 
1  Ik,  then  Xp  <  k(\/k)  =  1.  But,  Xp  must  equal 
one  so  p>k.  Clearly  the  lowest  possible  value  for 
p  is  1  Ik,  and  p  takes  this  value  only  when  all  p’s 
are  equal,  again  because  Xp  must  equal  one. 
Hence,  the  substitution  of  \jk  for  p  indicates 
correctly  a  state  of  complete  uncertainty -the  one 
for  which  the  same  score  as  an  omit  is  desired.  It 
remains  only  to  take  b  as  unity  to  yield  the 
standard  formula  score.  While  this  score  is  not 
quasi-reproducing,  neither  does  it  force  the 
student  to  overexpress  or  underexpress  his 
certainty  when  that  certainty  is  elicited.  The 
rather  surprising  result  here  is  that  if  a  linear 
scoring  system  were  to  be  used,  the  confidence 
elicited  should  not  be  scored  (a  and  A  are  zero). 
Further,  since  most  writers  agree  that  it  is 
important  to  inform  the  examinee  carefully  about 
the  scoring  system,  one  would  elicit  the  con¬ 
fidence  response  and  then  carefully  inform  the 
examinee  that  it  would  be  ignored!  It  is 
concluded,  therefore,  that  unless  one  is  prepared 
to  use  nonlinear  functions  of  the  confidence  ex¬ 
pressed,  one  should  not  attempt  to  introduce 
confidence  scoring. 


III.  THE  QUADRATIC  CASE 

More  useful  results  o'  ■:  in  the  case  of  the 
quadratic  scoring  function.  Here,  we  define 

X*)  =  bx2  +  cx  +  d 
and 

*(x)  =  Bx2  +  Cx  +  D 

to  obtain 

5  *  p(bx 1  +  cx  +  d)  +  ( 1  -  pXBx 2  +Cx  +  D ) 

and  choose  b,  B,  c,  C,  d,  and  D  so  that  S  is  at  a 
maximum  for  all  admissible  p,  and  so  that  XI/*)  = 
g(l/k)  =  0.  It  will  be  seen  that  these  requirements 
impose  five  conditions  on  the  six  constants, 
leaving  an  arbitrary  choice  of  a  sixth  condition. 
For  this  condition,  X  0  =  1  <s  chosen.  To 
maximize  5,  equate 

dSldx  =  2p/>;c  +  pc  +  (l  -p)  2 Bx  +  (1  - p)C, 
evaluated  at  the  point  p,  to  zero  to  obtain 
dS/dx|  p  =  p2(2Xh  -  fl)  +  Pic  +  2B  -  C)  +  C  =  0. 


Setting  coefficients  of  the  powers  of  p  to  zero, 
obtain 


Thus, 

and 


b  =  B,C=0,c  C+2B  =  Q. 


Ax)  =  bx2  -2b  +  d 


g(x)  =  bx 2  +  D. 
Then  XI Ik)  =  0  implies  that 

d  =  -b/k2  +  2b, 
and  g(l/A:)  =  0  implies  that 

D  =  -b/k2 . 


Thus, 

Ax)  =  b[x2  -2x-(l/*J)  +  (2/*>] 
and 


g(x)  =  b[x2  -(l/*2)]. 

Note  that 


dAx)ldx  =  6(2x  -  2) 

which  takes  on  the  opposite  of  the  sign  of  b  since 
2x  must  be  less  than  2  unless  x  equals  or  exceeds 
one  (which  it  cannot).  It  is  desirable  that  the 
derivative  of  X*)  with  respect  to  x  be  non-negative 
and,  therefore,  the  sign  of  b  should  be  negative. 

if  b  is  chosen  to  be  negative,  than  g(x)  will  be 
mnnotonically  decreasing  with  increasing  x\  and  if 
x  is  not  less  than  1  Ik,  the  reward  for  responding 
honestly  to  the  subjectively  most  probable  of  the 
correct  answers  will  always  be  greater  than  any 
other  course  of  action  provided  the  least  certainty 
the  examinee  is  allowed  to  express  is  complete 
uncertainty,  that  is  1  Ik.  This  caution  is  introduced 
because  under  certain  conditions  the  value  of  S 
will  be  greater  if  the  examinee  indicates  a  very 
small  subjective  probability  for  an  alternative  he  is 
virtually  certain  is  incorrect  than  if  he  marks  an 
alternative  he  is  moderately  sure  is  correct.  This 
possibility  is  to  be  avoided  because  it  is  relatively 
difficult  to  avoid  having  at  least  one  bad 
distractor;  it  will  be  shown  that  if  allowed,  the 
examinee  should  mark  the  wrong  distractor  with  a 
lower  subjective  probability  than  a  right  one, 
unless  he  is  pretty  sure  it  is  right.  To  show  this, 
suppose  that  the  examinee  is  certain  that  an 
alternative  is  incorrect  and  he  marks  it  zero.  Then 
his  payoff  is 

5f  =  0-X0)+  I  •*((>)  =  *(-!/*’) 

if  according  to  his  hypothesis  he  marks  the  wrong 
one  zero.  However,  if  he  is  to  mark  an  alternative 
that  has  a  chance  of  being  correct,  his  probability 
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may  be  as  low  as  1  /(*  -  1);  and  according  to  his 
hypothesis  his  payoff  would  be 


S„  =  —  A-^  +  (*  fa---) 

*  -  1  k-l  *  -  1 

-6 


*’(*  -  l)S 


Clearly ,  S.  -  — - — ,  S  and  is  less  than  S  . 
h  (*- l)2  e 

Therefore,  if  the  candidate  knows  the  payoff 
system,  he  should  in  this  case  indicate  that  the 
erroneous  distractor  is  incorrect  rather  than  make 
the  best  guess  he  can  about  which  alternative  is 
correct.  This  can  be  avoided  by  limiting  the  range 
of  responses  he  can  make  from  l  Ik  to  one  since  in 
this  range 

s=pfip)  +  (  1  -p)g(p), 

if  p  is  used  for  x,1  and  has  a  fust  derivative  equal 
to 

dS/dp  =  'fi—bfip  —  *') 
which  is  clearly  positive  if  b  is  negative. 

Finally,  for  the  sake  of  definiteness,  b  is  chosen 
so  that  fi 1)  equals  one.  That  is, 

l=ft(l  -2 -*"*+  2tc  ')  or  k2  =  -b(k  —  l)2 . 
Hence 

Xx)  =  k2(k‘2  +  2x  -x1  -2kMXk-l)"2  and 

g(x)  =  kI(*J-xJX*-l)'J- 


IV.  USING  DISCRETE  VALUES 

The  intent  of  the  foregoing  analysis  is  to  arrive 
at  a  scoring  function  which  is  reproducing  at  least 
in  the  sense  of  eliciting  an  honest  expression  of 
confidence  about  the  response  made  and,  further, 
one  which  requires  only  a  simple  response  from 
the  examinee  and  is  easy  to  process.  A  scoring 
system  which  requires  that  the  response  be 
recorded  as  a  number  for  one  or  all  alternatives 
requires  data  processing  steps  to  get  from  the 
recorded  response  to  a  machine-processable 
record.  These  steps  can  be  avoided  by  using  a 


'Since  it  is  known  that  the  scoring  system  is 
quasi-reproducing,  ir  is  proper  to  use  p  instead  of  x  as 
arguments  of /  and  g. 

2 The  task  of  the  subject  in  the  Dressel-Schmid  format 
is  like  that  of  method  B-l  of  de  Finetti  except  that  more 
confidence  levels  are  allowed.  The  scoring  rationale  here  is 
also  different. 


discrete  rating  system;  deFinetti  has  discussed  a 
number  of  such  systems.2  By  using  the  Dressel- 
Schmid  format  with  a  discrete  multilevel  con¬ 
fidence  rating  scale,  the  examinee  is  allowed  to 
make  a  very  simple  response  which,  through 
mark-sensing  or  optical  scanning,  is  directly  avail¬ 
able  for  quasi-reproducing  scoring  using  digital 
processing. 

Table  1,  which  could  serve  as  a  basis  for 
choosing  scores  for  discrete  responses,  contains  the 
scoring  system  for  common  numbers  of  alter¬ 
natives.  Note  that  in  this  table  the  scores  are  not 
defined  for  confidence  levels  below  l/k.  It  can  be 
seen  that  in  all  cases  fix)  has  a  positive  slope  and  a 
negative  acceleration.  Since  the  two  functions  take 
on  the  same  value  when  their  argument  equals  l/k, 
they  diverge  as  x  increases  as  does  the  risk  of 
expressing  an  increased  degree  of  certainty.  How¬ 
ever,  note  that  the  values  of  the  objective 
function,  S,  are  increasing  as  confidence  increases, 
so  the  examinee  can  indeed  expect  to  be  rewarded 
on  the  average  by  expressing  his  certainty  when  he 
feels  it.  Also  contained  in  Table  1  are  fix),  g(x), 
and  S  for  the  limiting  condition  of  k  equals 
infinity  (free  response  scored  right  or  wrong). 

It  is  felt  that  the  scoring  procedure  can  very 
well  be  approximate  so  long  as  some  provision  for 
expressing  confidence  is  made  and  the  scoring 
system  in  Table  1  is  roughly  reproduced.  Thus,  a 
method  for  obtaining  scoring  alternatives  is 
suggested;  (a)  Using  five  responses,  verbally 
describe  one  extreme  response  as  absolute 
certainty,  and  the  other  as  absolute  uncertainty. 
Then  the  sccuing  for  these  extremes  can  be  zero 
and  ten  (or  one  hundred),  if  the  response  is 
correct.  If  it  is  wrong,  the  scores  are  zero  and  ten 
(or  one  hundred)  times  the  entry  in  Table  1 
appropriate  to  the  number  of  distractors.  (b)  State 
verbally  that  the  middle  categories  represent  equal 
intervals  of  uncertainty  (or  certainty)  about  the 
answer.  If  the  response  is  a  “push,”  use  the  middle 
interval.  If  not,  use  one  of  the  other  two  to  show 
the  strength  of  certainty.  This  kind  of  language 
may  be  taken  as  justification  for  assigning  to  the 
categories  the  scores  from  one-sixth,  one-half,  and 
five-sixths  (the  category  midpoints  if  the  interval  is 
equally  divided  into  thirds)  of  the  distance  from 
complete  uncertainty  to  certainty. 

For  example,  if  a  true-false  test  (k  =  2)  is  given, 
the  lower  and  upper  category  boundaries  are  .5 
and  1 ,  respectively.  Then  the  two  middle  categoiy 
boundaries  are 

(l/fc)+*-tW*l,  .5+-!— £ 

3  3 
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Table  l.  Scorn*  for  Common  Numbers  of  Alternatives  as  a  Function 
of  Expressed  Confidence  Levels 


Percent 

Confluence 

Level 

noon) 

— 

Number  of  Alternatives 

2 

3 

4 

ft 

CX 

*00 

—Six) 

s 

*tx) 

s(x) 

s 

*(») 

— •<«! 

s 

*|x) 

— 9(X) 

S 

*(X) 

— S(x» 

S 

0 

0 

0 

0 

5 

0.1 

0.0 

0.0 

10 

0.2 

0.0 

0.0 

15 

0.3 

0.0 

0.0 

20 

0 

0 

0 

0.4 

0.0 

0.0 

25 

0 

0 

0 

0.1 

0.0 

0.0 

0.4 

0.1 

0.1 

30 

0.0 

0.05 

0.0 

0.2 

0.1 

0.0 

0.5 

0.1 

0.1 

33-1/3 

0 

0 

0 

0.2 

0.1 

0.0 

0.3 

0.1 

0.0 

0.6 

0.1 

0.1 

35 

0.0 

0.0 

0.0 

0.25 

0.1 

0.0 

0.3 

0.1 

0.0 

0.6 

0.1 

0.1 

40 

0.2 

0.1 

0.0 

0.4 

0.2 

0.0 

0.4 

0.2 

0.1 

0.6 

0.2 

0.2 

45 

0.3 

0.2 

0.0 

0.5 

0.25 

0.1 

0.5 

0.25 

0.1 

0.7 

0.2 

0.2 

50 

0 

0 

0 

0.4 

0.3 

0.1 

0.6 

0.3 

0.1 

0.6 

0.3 

0.1 

0.75  0.2S 

0.25 

55 

0.2 

0.2 

0.0 

0.5 

0.4 

0.1 

0.6 

0.4 

0.2 

0.7 

0.4 

0.2 

0.8 

0.3 

0.3 

60 

0.4 

0.4 

0.0 

0.6 

0.6 

0.2 

0.7 

0.5 

02 

0.75 

0.5 

0.25 

0.8 

0.4 

0.4 

65 

0.5 

0.7 

0.1 

0.7 

0.7 

0.2 

0.8 

0.6 

0.3 

0.8 

0.6 

0.3 

0.9 

0.4 

0.4 

70 

0.6 

1.0 

0.2 

0.8 

0.9 

0.3 

0.8 

0.8 

0.4 

0.9 

0.7 

0.4 

0.9 

0.5 

0.5 

75 

0.75 

1.25 

0.25 

0.9 

1.0 

0.4 

0.9 

0.9 

0.4 

0.9 

0.8 

0.5 

0.9 

0.6 

0.6 

80 

0.8 

1.6 

0.4 

0.9 

1.2 

0.5 

0.9 

1.0 

0.5 

0.9 

0.9 

0.6 

1.0 

0.6 

0.6 

85 

0.9 

1.9 

0.5 

0.95 

1.4 

0.6 

1.0 

1.2 

0.6 

1.0 

1.1 

0.7 

1.0 

0.7 

0.7 

90 

1.0 

2.2 

0.6 

1.0 

1.6 

0.7 

1.0 

1.3 

0.7 

1.0 

1.2 

0.8 

1.0 

0.8 

0.8 

95 

1.0 

2.6 

0.8 

1.0 

1.8 

0.9 

1.0 

1.5 

0.9 

1.0 

1.35 

0.9 

1.0 

0.9 

0.9 

100 

1 

3 

1 

1 

2 

1 

1 

1-2/3 

1 

1 

1-1/2 

1 

1 

1 

1 

*Whcre  decimals  are  given,  rounding  is  to  die  neatest  low  older  position;  figures  without  decimals  are  exact. 

Table  2.  Approximate  Scotea  for  Responses  to  Confidence  Items 

on  Dressei-Schmid  Format 

Credit  if  Rlftit 

Loss  If  Wrong 

Category 

a* 

2 

2 

6 

5 

a* 

2 

2 

4 

S 

04 

Absolutely  Certain 

100 

10 

10 

10 

100 

300 

20 

17 

15 

100 

Certain 

95 

9 

9 

9 

88 

225 

14 

12 

11 

43 

Middle  Certain 

75 

7 

7 

7 

75 

125 

7 

5 

5 

25 

Somewhat  Uncertain 

35 

3 

3 

3 

28 

20 

2 

2 

1 

2 

Completely  Uncertain 

0 

0 

0 

0 

0 

0 

0 

0 

0 

0 

*100- point  scale  used  to  avoid  duplication  from  rounding. 


and 

(1/fc)  +  2(i-JL*.L«])  ,.5  +  »U— jgl 

vi/  3  3 

Then  ,  the  category  midpvnts  are 

.5  +  4JL,.5+I^,and.5 

6  2  6 

Table  2  gives  the  tabled  weigt.i.  The  true-false  and 
free  response  scores  are  given  on  a  correct  score 


scale  of  0  to  100,  rather  than  ten,  to  avoid 
identical  weights  for  different  responses.  The  table 
entries  can  easily  be  displayed  on  an  answer  sheet 
or  provided  to  an  examinee  as  ancillary  material. 
Finally,  the  table  can  be  adapted  to  a 
four-alternative  answer  sheet  by  instructing  the 
examinee  to  omit  the  item  if  he  has  no  preference 
among  any  of  the  alternatives. 
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V.  PERSPECTIVE 


Confidence  testing  seems  to  hold  promise  for 
the  person  who  is  concerned  about  certain 
anxiety-producing  aspects  of  tests  which  use  the 
usual  formula  scoring.  Since  it  is  reasonable  to 
assume  that  there  is  a  difference  in  knowledge 
between  persons  who  are  confident  that  wrong 
answers  are  correct  and  those  who  express  wrong 
responses  diffidently,  it  is  certainly  of  interest  to 
find  some  way  to  improve  the  task  of  the 
examinee,  as  well  as  that  of  the  one  who  must 
interpret  his  performance.  The  present  paper 
presents  a  way  of  accomplishing  confidence  testing 
which  is  considered  to  be  relatively  easy  to  use  and 
which  has  an  appealing  rationale. 

At  the  same  time,  however,  the  author  does  not 
suggest  that  confidence  testing  in  any  known 
format  would  necessarily  be  anxiety  reducing  in  all 
situations;  neither  is  it  suggested  that  use  of  the 
method  described  would  result  in  anxiety 
reduction  in  any  given  situation  at  the  present 
time.  It  has  also  been  pointed  out  that  confidence 
testing  is  not  expected  to  make  major  increases  in 


reliability  or  validity.  In  fact,  Swineford  (1938, 
1941)  presents  evidence  that  the  tendency  to 
claim  extra  credit  under  conditions  of  risk  is  quite 
unrelated  to  other  variables.  He  suggests  further  a 
possible  contamination  of  scores  based  on 
confidence  techniques  due  to  irrelevant 
personality  trends. 

When  converting  from  a  standard  multiple- 
choice  test  to  a  confidence  format,  one  should  at 
least  consider  the  assessment  of  a  response  style 
with  respect  to  risk  in  order  to  determine  whether 
some  allowance  should  be  made  for  that  style. 
However,  response  styles  and  personality  factors 
may  be  operative  under  current  testing  modes  as 
well  as  under  confidence  testing.  It  is  not  that  one 
is  “right”  but,  rather,  that  both  may  be  used,  and, 
when  they  are,  possible  moderation  by  personality 
scores  could  well  be  considered.  And  when  such 
consideration  is  given,  the  method  presented 
herein,  with  its  comparative  simplicity  of  use  and 
scoring,  is  recommended. 
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